Papers with progressive module
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to compress neural networks by progressive module replacement is proposed . a number of techniques have been proposed to compress pretraining and fine-tuning models . |
| Approach: | They propose a model compression approach that divides BERT into modules and builds their compact substitutes. |
| Outcome: | The proposed approach outperforms existing knowledge distillation approaches on GLUE benchmark . it is based on a model that divides the original BERT into several modules and builds their substitutes . |
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation. |
| Approach: | They propose two new metrics that measure how closely a compressed model mimics the original model. |
| Outcome: | The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher). |